#22002

This set of changes is snapped off from #21938, where it introduces a way to relate to an s3 relation and get the correct credentials instead of using the ones on controller config. By utilising the [juju-controller charm](juju/juju-controller#107) to drive the changes, we'll be able to ensure that we benefit from the latest charm infrastructure. 

The changes only target the state code at the moment, with services coming in a later patch.

## QA steps

This breaks s3 charm integration, as we require all service changes to be in place. 

## Links

<!-- Link to all relevant specification, documentation, bug, issue or JIRA card. -->

<!-- Replace #19267 with an issue reference or link, otherwise remove this line. -->
**Issue:** Fixes #19267.

<!-- Place JIRA number in both places below. -->
**Jira card:** [JUJU-](https://warthogs.atlassian.net/browse/JUJU-)

Adds s3 endpoints to update the objectstore backend to s3. This is
broken off from[1], as that was rather large, it's better to break
the work down.

This follows both the metrics and tracing endpoint. It will integrate
with the controller charm[2]. Ultimately this will allow the charm
integrations drive the controller like a real application.

Adds endpoints and calls service methods with the JSON input. Currently
the removal of a relation will not be permitted, this will change in
the future.

 1. juju#21938
 2. juju/juju-controller#107

Adds s3 endpoints to update the objectstore backend to s3. This is
broken off from[1], as that was rather large, it's better to break
the work down.

This follows both the metrics and tracing endpoint. It will integrate
with the controller charm[2]. Ultimately this will allow the charm
integrations drive the controller like a real application.

Adds endpoints and calls service methods with the JSON input. Currently
the removal of a relation will not be permitted, this will change in
the future.

 1. juju#21938
 2. juju/juju-controller#107

…ration

#22267

Adds s3 endpoints to update the objectstore backend to s3. This is broken off from[1], as that was rather large, it's better to break the work down.

This follows both the metrics and tracing endpoint. It will integrate with the controller charm[2]. Ultimately this will allow the charm integrations drive the controller like a real application.

Adds endpoints and calls service methods with the JSON input. Currently the removal of a relation will not be permitted, this will change in the future.

 1. #21938
 2. juju/juju-controller#107

## QA steps

Regression tests should be enough for this: #22146

Otherwise a simple bootstrap should suffice as nothing is wired up:

```sh
$ juju bootstrap lxd test
```

…ig-s3-integration

#22304

Removes controller config for s3 integration, instead the s3 credentials will be driven by the controller charm[1]. This is broken off from[2], as that was rather large, it's better to break the work down.

This starts to lay the ground work for implementing all the methods required for getting a file backed controller to an s3 backed controller. This should be the penultimate PR for completing the tasks. Further changes to allow s3 to s3 bucket transfer will be done in the future, but not now.

In order to make life easier when implementing the drainer, the controller config is just removed. Everything should be driven off from the controller object store service.

> [!NOTE]
> As this is controller only config, and we don't allow controller migrations, we can deprecated the old config gracefully.

 1. juju/juju-controller#107
 2. #21938


## QA steps

Regression tests, though the file to s3 backed storage draining is broken in this PR, until the subsequent PR lands.

```sh
$ juju bootstrap lxd test
```

#22380

~~Requires #22304

This is one of the last change sets that allows the draining from file object store to s3 backed object stores. We don't currently allow s3 to another s3, as there are some concerns around this. For now it's restricted until we iron out those issues.

In order to make life easier when implementing the drainer, the controller config is just removed. Everything should be driven off from the controller object store service.

----

A lot has changed in this PR because we can reduce a lot of complexity from the prior object-store values in controller-config. This includes trying to push as much of the calls into singular transactions, which should prevent any potential races.

----

As we'll need to follow-up with s3-to-s3 draining, but this effectively removes a potential issue with exposing keys to the user.

 1. juju/juju-controller#107
 2. #21938

## QA steps

### S3 setup

I'm using minio here, but that's only because of familiarity and we should probably use something that isn't no longer maintained.

```sh
docker run -d \n -p 9000:9000 \n -p 9001:9001 \n --user $(id -u):$(id -g) \n --name minio1 \n -e "MINIO_ROOT_USER=ROOTUSER" \n -e "MINIO_ROOT_PASSWORD=CHANGEME123" \n -v /home/jenkins/.local/minio/data:/data \n quay.io/minio/minio server /data --console-address ":9001"
```

```sh
$ docker exec -it <container> bash
$ mc alias set foo http://localhost:9000 ROOTUSER CHANGEME123
$ mc admin info foo
$ mc admin accesskey create foo
# Note down the access keys
```

### Build controller charm

Checkout the following juju/juju-controller#107 and then pack it for use.

```sh
$ charmcraft pack
$ mv juju-controller_ubuntu@24.04-amd64.charm <juju location>
```

### Bootstrap

```sh
$ juju bootstrap lxd src --bootstrap-base=ubuntu@24.04 --controller-charm-path=./juju-controller_ubuntu@24.04-amd64.charm --build-agent
```

Add another model so we can test that:

```sh
$ juju add-model foo
$ juju deploy juju-qa-test
```

Add the s3-integrator so it can drive the draining:

```sh
$ juju switch controller
$ juju deploy s3-integrator
$ juju integrate controller:s3-backend s3-integrator
$ juju config s3-integrator endpoint=<url to minio or S3 compatible>
$ juju run s3-integrator/leader sync-s3-credentials access-key=<access key> secret-key=<secret key>
```

This should start the draining - check `juju debug-log -m controller` for errors.

Alternatively check the db:

```sh
$ juju ssh -m controller 0
$ juju_db_repl
repl (controller)> SELECT prdesc FROM object_store_backend
uuid life_id type_id updated_at
653813f9-2896-5332-8cbe-629a337a56a3 2 0 2026-05-07 10:40:57.846945418 +0000 UTC
f080e651-7c56-4242-8773-260110069191 0 1 2026-05-07 10:40:57.846945418 +0000 UTC
```

We're looking for the life of the file (type_id: 0) to be 2, ensuring it's completed and it's dead.

### Verification

Check the buckets:

```sh
$ mc ls foo
[2026-05-07 10:40:59 UTC] 0B juju-7f618674-29f6-4781-88fb-6e61c6339c42/
```

Check the controller file based object stores are empty:

```sh
$ juju ssh -m controller 0
$ ls -la /var/lib/juju/objectstore
```

Check that we can move into HA (there is a [bug moving into HA](juju/juju-controller#117)):

```sh
$ juju add-unit -m controller controller -n 2
```

Check you can deploy other things:

```sh
$ juju switch foo
$ juju deploy ubuntu
```

…-on-db

#22547

This change removes the remaining object store drainer dependency on
`agent.conf`.

The object store backend and draining lifecycle are already DB-backed. This PR
finishes that cleanup by removing the leftover agent-config plumbing from the
drainer path, so `objectstoredrainer` no longer reads or mutates `agent.conf`
and no longer carries object-store-type state internally.

## Why

`agent.conf` was still carrying stale object store state that no longer matched
the real ownership model.

Object store backend selection and draining are controller-wide concerns.
Keeping residual backend-type state in agent config meant the drainer still had
to reconcile local config with DB-backed state, even though the database is the
authoritative source.

This change removes that split responsibility and leaves the drainer as a
consumer of existing DB-backed draining state.

## Previous behavior

Even after the object store transition flow moved to DB-backed services, the
drainer still had logic tied to `agent.conf`:

- the manifold read agent config during startup
- startup logic could reconcile object store type through agent config
- the worker still carried object-store-type state internally
- draining completion still updated `agent.conf`
- agent config still exposed and serialized `ObjectStoreType`

So while backend/draining state was already DB-driven, the drainer still had a
leftover config-state path.

## New behavior

The drainer now operates only on DB-backed draining state:

1. startup reads agent config only for normal agent runtime data such as
 `DataDir`
2. `objectstoredrainer` watches DB-backed draining state
3. when draining is active, it locks down, drains controller/model data,
 flushes object store workers, and marks draining completed
4. no object store backend state is read from or written to `agent.conf`

So the drainer is now a pure draining-phase consumer.

## Result

After this change:

- `agent.conf` is no longer involved in the drainer flow
- object store backend/draining state remains DB-owned
- `objectstoredrainer` has a narrower and cleaner responsibility
- the remaining object store migration path is easier to reason about

## Checklist

- [x] Code style: imports ordered, good names, simple structure, etc
- [x] Comments saying why design decisions were made
- [x] Go unit tests, with comments saying what you're testing
- [x] [Integration tests](https://github.com/juju/juju/tree/main/tests), with comments saying what you're testing
- [ ] ~[doc.go](https://discourse.charmhub.io/t/readme-in-packages/451) added or updated in changed packages~

## QA steps

Set up minio as S3 compatible store:

```sh
❯ mkdir -p /home/jenkins/.local/minio/data
❯ docker run -d \n -p 9000:9000 \n -p 9001:9001 \n --user $(id -u):$(id -g) \n --name minio1 \n -e "MINIO_ROOT_USER=ROOTUSER" \n -e "MINIO_ROOT_PASSWORD=CHANGEME123" \n -v /home/jenkins/.local/minio/data:/data \n quay.io/minio/minio server /data --console-address ":9001"1

❯ docker exec -it minio1 bash
❯ mc alias set foo http://localhost:9000 ROOTUSER CHANGEME123
❯ mc admin info foo
❯ mc admin accesskey create foo
# Note down the access keys
```

**Build controller charm**
Checkout the following juju/juju-controller#107 and then pack it for use.

```sh
❯ charmcraft pack
❯ mv juju-controller_ubuntu@24.04-amd64.charm <juju location>
```

**Bootstrap**

```sh
❯ juju bootstrap lxd src --bootstrap-base=ubuntu@24.04 --controller-charm-path=$GOBIN/juju-controller_ubuntu@24.04-amd64.charm --build-agent
❯ juju add-model foo
❯ juju deploy juju-qa-test
```

**Drain:** 

Add the s3-integrator so it can drive the draining:

```sh
❯ juju switch controller
❯ juju deploy s3-integrator
❯ juju integrate controller:s3-backend s3-integrator
❯ juju config s3-integrator endpoint=http://172.17.0.1:9000
❯ juju run s3-integrator/leader sync-s3-credentials access-key=<access key> secret-key=<secret key>
```

Check the logs:

```sh
❯ juju debug-log --replay -m controller | rg drain
machine-0: 16:53:48 DEBUG juju.worker.dependency "object-store-drainer" manifold worker started at 2026-06-02 14:53:48.17802061 +0000 UTC
machine-0: 16:53:48 INFO juju.worker.objectstoredrainer object store is not draining, unlocking guard
machine-0: 16:53:52 DEBUG juju.worker.dependency "user-secrets-drain-worker" manifold worker started at 2026-06-02 14:53:52.353946331 +0000 UTC
machine-0: 16:57:33 INFO juju.worker.objectstoredrainer object store is draining, locking guard
machine-0: 16:57:33 INFO juju.worker.objectstoredrainer draining controller agent binaries
machine-0: 16:57:33 INFO juju.worker.objectstoredrainer runner "objectstore-drainer" starting worker "controller"
machine-0: 16:57:33 WARNING juju.worker.objectstoredrainer drain attempt 1/10 for "controller" failed: no session available, retrying
machine-0: 16:58:04 INFO juju.worker.objectstoredrainer drain worker for controller agent binaries completed
machine-0: 16:58:04 INFO juju.worker.objectstoredrainer draining model "c31c1380-1be6-4ded-86b8-e3d7ec6954e1"
machine-0: 16:58:04 INFO juju.worker.objectstoredrainer draining model "d290d323-7b1e-4eda-8e8a-3a1fceb84499"
machine-0: 16:58:04 INFO juju.worker.objectstoredrainer runner "objectstore-drainer" starting worker "d290d323-7b1e-4eda-8e8a-3a1fceb84499"
machine-0: 16:58:04 INFO juju.worker.objectstoredrainer runner "objectstore-drainer" starting worker "c31c1380-1be6-4ded-86b8-e3d7ec6954e1"
machine-0: 16:58:04 INFO juju.worker.objectstoredrainer drain worker for model "c31c1380-1be6-4ded-86b8-e3d7ec6954e1" completed
machine-0: 16:58:04 INFO juju.worker.objectstoredrainer waiting for 1 more drain workers to complete
machine-0: 16:58:04 INFO juju.worker.objectstoredrainer drain worker for model "d290d323-7b1e-4eda-8e8a-3a1fceb84499" completed
machine-0: 16:58:04 INFO juju.worker.objectstoredrainer object store draining completed successfully
machine-0: 16:58:11 INFO juju.worker.objectstoredrainer object store is not draining, unlocking guard
```

Check the db:

```sh
❯ juju ssh -m controller 0
❯ juju_db_repl
repl (controller)> SELECT prdesc FROM object_store_backend
uuid life_id type_id updated_at 
653813f9-2896-5332-8cbe-629a337a56a3 2 0 2026-06-02 14:57:26.502027555 +0000 UTC
7cb2a3fa-c8ab-45a9-82b4-7f16789ae21d 0 1 2026-06-02 14:57:26.502027555 +0000 UTC

repl (controller)> SELECT prdesc FROM object_store_drain_info
uuid phase_type_id from_backend_uuid to_backend_uuid 
45a12f7f-c6ad-445b-8d94-a6907fa7a0c2 3 653813f9-2896-5332-8cbe-629a337a56a3 7cb2a3fa-c8ab-45a9-82b4-7f16789ae21d 
```


## Documentation changes

## Links

Relates to: #22380

**Jira card:** [JUJU-9721](https://warthogs.atlassian.net/browse/JUJU-9721)


[JUJU-9721]: https://warthogs.atlassian.net/browse/JUJU-9721?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ